Physics Engine Development 



Hello! 

My name is Sergiy Migdalskiy and I work at Valve with Dirk Gregorius. I helped port, 
develop and optimize our in-house physics engine, Rubikon. Also, I have implemented a 
physics engine for Uncharted: Drake's Fortune on PS/3. 












Physics Engine Development 


♦ 


A , 






While this talk is about physics development, I'd like to emphacise that tools and techniques 
I'm describing are general purpose. They are probably applicable to any complex system 
written in C++. 
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Visualization 



Run Playing_with_Physics.nnov 
This is what I do the whoie day long. 

I play with physics, throw ragdoils into sandboxes, and such siiiiness. 

(wait) Here I'm piling up some ragdoiis, it's looking good, untii.. 

I notice one of the ragdoiis, the very first, doesn't seem to coilide with anything. 

I'm testing some more. I make sure the other ragdoiis work fine. Then I notice something. 
Here it is - the collision fiags on the very first ragdoll are set to not coiiide with anything. 
Now the mystery is solved, and I go on to other tests. 






We started working with Dirk by porting his rigid body engine to Valve's framework. I 
knew Qt and I like playing with stuff I'm working on, for fun and to make it easier. 
So I implemented a little IDE for our physics project. 

It was incremental. Every time something took more than 30 seconds seconds to 
look up in the debugger, I made a widget that makes it a split-second. 

We had physics running in an isolated tool with buttons, sliders and 3D window with 
shadows, the one you've just seen, which was pretty cooi. And it restarted in 
seconds. We call it the Physics Testbed, or the Physics Debugger. Not to be confused 
with Visual Studio debugger. 
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Collision in Game 


Physics + Game = Slow Dev Cycle 


But there comes the time for every game physics engine to be put into an actual 
game. 

Once we started integrating physics with the game, we had to debug it inside the 
game. So we lost our fast-iteration, nice Ul. 

Every engine has in-game visualization of physics. So we make do. 

It has some serious drawbacks, though. 

. You can't easily use mouse cursor for clicking collision or Ul. 

. You cannot fly around easily, you have to implement a special camera for it. 

. You need to render inside game engine, which can be pretty tricky. 

. You have to implement a special pause that pauses absolutely everything 

while you're examining something. 

• It's non-trivial to implement rich Ul, like you can do with Qt. 
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Physics Ul 


Collision in a Separate Window with Ul 



Physics Testbed was a great Ul with a bunch physics-centric widgets. It was a great 
help already in making physics work. I felt I can reuse it in the game integrated 
physics. 

But how do we display in-game physics world in a separate window? 
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Extra Window 


Game with extra Window 




I briefly entertained an idea to open an extra Qt window in the game process. I find 
it: 

• inconvenient to implement 
. It's intrusive to the main game loop 

. does not work when the game is crashed or stopped on a breakpoint 

. You have to implement “full pause” in the game, both on server and client 

So I decided against in-process window. This means I could just reuse the existing 
separate tool, our Physics Testbed. 
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Out-of-Game Visualization 



To visualize in external process, you need to send data to it. 

The hardest part about sending physics data to external app is serialization. 

But if you can properly and fully serialize your world state into a stream of bits and 
send it over the wire, it becomes very straightforward to visualize your physics in 
a separate app. 

Playing with physics in external app is a nice bonus feature. 
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Serialization 



So serialization is the biggest obstacle here. But once we solve it, it becomes very 
useful: 

- we can save and look at something later 

- we can implement nice data and time profilers 

- we can record and replay bugs 

Once you can serialize, it is trivial to stream to external app. 
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(Un)Serialization 



By the way, of course, serialization works two ways. The external app has to 
somehow turn our stream of bits into data structures.. 

So How do we serialize? Physics engine is complex. There are so many classes and 
data structures in there! Serializing pointers is hard. How do we build software 
that doesn’t miss anything, and then reconstructs everything faithfully. And 
hopefully it updates itself every time we add or remove a field in a class? 

Definitely not! by writing serialization by hand. 
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Generating C++ 


Data Declaration* 


/ 


\ 


\ 


Serializer Code 


C++ Classes 


Other 


* Data Declarations in easy-to-parse language (not C++) 


. One way to solve it is to describe your data structure in non-C++ (like 
protobufs), and generate C++ code from that. 

. The description has to be easy to parse 

We already had C++ code to start with, and the data structures were not easy to 
express in something like protobufs. I'd have to struggle to define the language 
and then struggle to express our nontrivial C++ data structures. 

If only we use C++! Then there is no need to remember the new language syntax. 
Anybody can come in and modify the code without learning an extra language 




















ANTLR 


C++ Classes 



ANTLR is a parser generator and was the first thing I used. It has a whole IDE 
for debugging grammars. It's very powerful. 

In house, we use ANTLR for our reflection API. We only parse a subset of C++ 
specification. I took that parser and I still use it to generate clean public 
interfaces to our physics engine. They take care of type conversions, they log 
external activity and do some other things. 

ANTLR is not ideal for parsing C++, though. You have to maintain C++ 
grammar, Java is its native language so it's the easiest target to generate 
parsers. It's fast, but an optimized C++ parser is faster. 




















Generating C++ from C++ 


C++ Classes 


Serializer Code 


We can analyze the engine's header files to generate serialization code. 

That's what we would do if we wrote the serialization by hand. 

But instead of doing it by hand we can parse C++ in a utility and spit out C++ which 
will serialize our data. 

But parsing C++ is extremely hard. Or is it? 
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Clang to the Rescue! 


C++ Classes 



Cla 

ng 


Serializer Code 


Clang is an open-source c-i-i- compiler. In some ways it's a direct competitor of 
GCC, but with much more readable source code and liberal license. 

Clang makes C-i-i- parsing a piece of cake. It's written in C-i-i- itself, and it just 
works. 

We can use all our existing code as input, convert cpp and header files into 
easy-to-use AST and spit out the serializer routines. 

One bonus from using Clang is this improved C-i-i- compliance of our code. We 
found some bugs, and prepared the code for Linux. It's really useful to compile 
your code with another compiler. 

















Clang AST 


Source code 


class CRnCapsuleShape : 
public CRnShape 

publ ic: 

private: 

Vector m_vCenter[ 2 ]; 
float m_flRadius; 
auto.serialize; 


Abstract Syntax Tree 



I mentioned the term AST. It means Abstract Syntax Tree. 

It's a tree that describes everything there is to know about your source code. 
This is how it looks, schematically. For every class, you'll have a node. For every 
member, its type or size, or annotation, or anything you can think of you'll 
probably get a node of some type. 
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Code Generation 


C++ 


Clang 


C++ 


#inc1ude 


void CRnCapsuleShape: :Serialize( 



CRnSerializer *pOut ) const 

class CRncapsu!eshape : 


{ 

public CRnShape 


CRnShape: : Serialize( pOut ); 

{ 


pOut->writeBuil tin<float>( m_flRadius ); 

pub1 ic: 


for( int nElement = 0; 

private: 


nElement < 2; 


nElement ) 

vector m_vCenter[ 2 ]; 


{ 

float m_flRadius; 


::Serialize( pout, 

auto_serialize; 

}; 


m_vCenter[nElement] ); 


In effect, we are transforming C++ code into additional C++ code 

It's very tempting at first to put code generation as a bunch of printf() statement 

right into the your parser 

I decided to resist the urge, and I consider myself lucky I did. It would be 
horribly slow to iterate on the generated code with printfs. 


16 



























The Structured Approach 


C++ 


Clang 


json 


#i nc1ude "..." 

class CRnCapsu!eShape : 
public CRnShape 

pub1 ic: 

private: 

vector m_vcenter[ 2 ]; 
float m_flRadius; 
auto_serialize; 


"CRnCapsuleShape" : { 
•’fields" : { 

"m_flRadius" : { 


"typeName" : 

} > 

"m_vCenter" : { 

"float", 

"className" : 

: "Vector", 

"arraySize" : 
}, ^ 

"bases" : [ 

"CRnShape" 

2, 


Parse, then Generate. Decouple these two stages, and unless your project is trivial, 
you'll be rewarded. 

I'm distilling the huge AST of the physics engine into a very concise json file. Json is 
a human readable format that is widely used nowadays. 

After adding features for a year, my Visitor class in the code that uses Clang that just 
prints out the json with relevant information is about 1000 lines long, and it would 
be incomprehensible if I generated code there. 
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Json? 


C++ - Clang — json — 

? - C++ 


So^ I'm taking my C++ code, let Clang parse it and spit out Json. 

But I ultimately need my serializer and potentially other useful tools to be written in C++ 

This begs the question: how do I generate C++ code from Json? 
























StringTemplate (to the rescue)! 


C++ 

- 

Clang 

— 

json 

— 


StringTemplate ^ C++ 


Fortunately, there is no need to reinvent the wheel, because it was invented many times over 
before us. There are a lot of text template libraries, and languages. They are widely used and 
freely available. 

I chose StringTemplate. It's a text library that's very easy to use, and it's very well suited to 
code generation. It's declarative style, which is appealing for this kind of task. 
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StringTemplate (to the rescue)! 


C++ 

- 

Clang 

— 

json 

— 


StringTemplate ^ C++ 


XML 


XSLT 


SQL PHP 


You could use something else. In fact, if you're very familiar and comfortable with, say, PHP, 
you should absolutely use it. 

You could also use XML instead of Json. In fact, standard Clang tools can output full abstract 
syntax tree in XML directly. XSLT is a powerful transformation language for XML. 
StringTemplates and Json are much easier to learn and much more human-readable. That's 
why I personally chose them. 

There are many ways to configure this pipeline. I'm just describing what worked for me. 
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StringTemplate 

C++ -► Clang —► 

json —► StringTemplate —► 

C++ 





void <name>:: Serialize( 

CRnSerializer *pOut ) const 

<c1ass.bases : 

{b 1 <b>: :Seria1ize( pOut ); }> 

<class. fields. keys, class. fields. values: {k, 

V I <field( name = k, props = v )> }> 






StringTemplate looks like this. The tool itself is written in java, but there's no need to know 
Java to use it. All my serialization, unserialization and memory statistics code is generated 
from a 400-line template. These are a few of those lines. They auto-generate the code to 
serialize most classes in our physics engine. 

You feed it json file, it spits out C-i-i- file. Quite simple. 
































Code Generation: Sources 


C++ 


Clang 


json 


StringTemplate 


C++ 


#include 

class CRnCapsuleShape : 
public CRnShape 

pub1 ic: 

private: 

Vector m_vCenter[ 2 ]; 
float m_flRadius; 
auto_serialize; 


So^ to recap. 

We start with the source code of the engine - it's arbitrary C++, potentially annotated. For 
instance, you can annotate only the classes you want serialized. And exclude some fields 
from serialization. And add a member to call after serialization. 
































Code Generation: Parser 


C++ 


Clang 


json 


StringTemplate 


C++ 



clang:iCXXMethodDecl * FindMethodswithNameC const clang::CXXRecordDecl *pRecord, 

const char *pName ) 

for( auto itMethod = pRecord->method_begi n(), 
itMethodEnd = pRecord->method_endC) ; 
itMethod != itMethodEnd; ++itMethod ) 

{ 

if( clang::ldentifierlnfo *pldlnfo = ( *itMethod )->getldentifierC) ) 
ifC pldlnfo->getName() == pName ) 
return *itMethod; 

} 

} 

} 

return NULL; 

} 


The parser/Clang part is moderately complicated. I wrote the first working version 
within a couple of weeks, working on It part-time. But then I fiddled with It for well over 
a year. It handles all the special and edge cases we have in our engine. It's not a 
magic bullet, though. It will not parse anything and everything In the known universe, 
but it'll parse and describe any exotic data structure in our engine. I have to change it 
only very rarely. In fact, the last change I made was very minor. And it was over 4 
months ago. 

This slide shows how to look for a method with a given name in class. It's pretty 
straightforward to extract information once you start writing the code. 

The best source of information for me was the online Clang docs, especially their 
class diagram. It's very logical and educational. Also, Clang is very readable, so a lot 
of stuff becomes clearer after you debug it. 
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Code Generation: Parser 


C++ 


Clang 


json 


StringTemplate 



"CRnCapsuleShape" : { 

•'fields'* : { 

"m_flRadius" : { 

"typeName" : "float", 

y 1 

"m_vCenter" : { 

"className" : "Vector", 
"arraySize" : 2, 

}, ^ 

"bases" : [ 

"CRnShape" 

}, 


C++ 


The generated json file looks like this. I already explained the meaning of this snippet 
before. 
































Generated Code 


C++ 


Clang 


json 


StringTemplate 


C++ 



void <name>:: Serialize( 

CRnSeria!izer *pOut ) const 

<class.bases : 

{b I <b>:: Serialize( pOut ); }> 

<class .fields. keys , cl ass .fields. values : 
{k, V I <field( name = k, props = v )> }> 


void CRnCapsuleShape: :Serialize( 

CRnSerializer *pOut ) const 

CRnShape: : Serialize( pOut ); 
pOut->writeBuil tin<float>( m_flRadius ); 
for( int nElement = 0; 

nElement < 2; 
nElement ) 


{ 


::Serialize( pOut, 

m_vCenter[nElement] ); 


To remind, this is what the string template looks like, and this here is the resulting 
generated C++ code 

The template goes through all data members of the class and generates code for 
each of them. 

The template is smart enough to know where I have arrays, templates, pointers to 
follow and so on. 

All the non-plain data elements are recursively serialized by other overloaded 
serialization functions. Some of them methods, some of them free functions. I 
especially prefer free functions for simpler types, containers and third party classes 
where I don’t want to touch the header where those classes are declared. 
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Once you complete these 5 easy steps, you'll have yourself one wholesome serializer. It will 
be self-maintaining. You will forever be able to visualize your physics in all detail. 

Sometimes we don't even know something is wrong. Some bugs ship and go unnoticed for 
years. This will never happen to you again. 

Well..., I can't make guarantees... But I really hope I convinced you that visualizing stuff is 
great. Having serializer auto-magically generate itself makes everything even better. 

But we have quite a bit more to discuss before we wrap up. 
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Record and Replay 



=Play= Top_Heavy_Lamp.wmv 

Here's an example of one workflow when you have full serialization. 

I noticed something weird with a lamp here. It was always falling down. Lamps are 
supposed to stand straight, and this one was always on its side. 

So I snooped and looked at the lamp. 

(wait) When I figured out why it's not standing (because its center of mass is too 
high), I just sent out an email to an artists - and I attached the file he can play 
with to prove my words. It's quite intuitive. 




Clang's “Hello World!” 


See ClangCheck.cpp 

class CDumpAction : public dang: :ASTConsumer 

llvm: : raw_fd_ostream &m_log; 
public: 

CDumpActionC 11vm: : raw_fd_ostream &fd ):m_log( fd ) 

} 

virtual bool HandleTopLevelDecl ( clang::DeclGroupRef DG ) 

for ( clang: :DeclGroupRef: : iterator it = DG.beginO, 
itEnd = DG.endC); it != itEnd; ++it ) 

clang::Decl *pDecl = *it; 
pDecl ->dumpXML( m_log ); 

return true; 

} 

}; 


Here is the Clang's “Hello World” program relevant to this use case is called ClangCheck and it is a 
great starting point to write your own tool. It's part of the Clang distro, and I urge you to have a 
look at it. 

It uses visitor pattern. For every top-level declaration in your source file, such as a class or function 
definition, or a typedef, or a global, you get a callback. In that callback, you can drill down the 
AST. 

This self-contained example is dumping those declarations as XML to a log. If XML is your thing and 
performance is not the most critical issue, this here just may be your parser. 
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Typically, all our class declarations are contained in headers. 

It's enough to parse one .cpp file that #includes all the relevant headers for serialization. 
There's no need to parse the whole engine, although there may be other useful tools that 
would need it. 

To parse just one file, you'll need to create a compilation database file with cpp file name 
and compile options. 




























Clang Compilation Database 


"directory”: , 

"command" :"clang -fsyntax-only -fms-extensions -fms-compatibility -l../public/include 
rnserialize.cpp", 

"fi1e" :"rnserialize.cpp' 


Here's an example of compilation database. The most interesting options are highlighted 
here. 

Compilation database is like a makefile, or vcproj. If you don't want to do anything non¬ 
standard, you can simply generate a .json file with all the information and give it to clang. 

We use a tool to generate VCPROJ files, and we changed that tool to also spit out 
compilation databases. 
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Clang Bootstrap Cheatsheet 


class MyAction: public clang:lASTConsuner 

publ ic: 

virtual bool HandleTopLevelDecl ( clang:lOeclGroupRef dg ) 

for ( clang::DeclGroupRef: : iterator 

it = DG.beginO, itEnd = DG.endO; it != itEnd; ++it ) 

clang::CXXRecordDecl *pRecord = 

llvw: :dyn_cast<clang: :CXXRecordDecl>( *it ); 
// do something with it... 

return true; 

} 

}; 


class MyFactory 

public: 

clang::ASTConsumer 

* ne^ASTCon s ume r () 

{ 

return 

new MyActionC ); 

} ^ 




pCompilationDb = clang: ; tooling : :DSONCompilationDatabase: :loadFromFile( strJsonDb, errorMessage ); 
files = p3sonDb->getAll Fi1es() ; 

clang: : tooling :: ClangTool Tool( *pCompilationDb, files ); 

MyFactory factoryC pDsonDbPath ); 

int nError = Tool. run ( clang: : tooling : :newFrontendActionFactory( &factory ) ); 
llvm: :errs().flush() ; 
llvm: :outsO .flush() ; 


And here is a little Cheatsheet. In your parser tool, that you carefully link with all the clang 
libraries, what do you do first? 

Start with implementing ASTConsumer, make a factory class for it, and then in your main() 
function, load compilation database and then run an instance ClangTool. It's all here in 
this slide. 

One thing to remember: you may have to flush std out and std error. 
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Box Cast 



Before going further, I'd like to clarify what I mean by box cast. 
We often use it in Source engine. It's like raycast, but with a box. 








Contact 



Just take a box, move it linearly until you hit something. 

At that point, you sometimes create a contact point for the solver. 

We use it for player movement a lot, so debugging it is a common thing to do. 
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Visualizing Physics In Game 



=Run= Player_Movement.mov 

Let's see what we've got for our efforts so far. 

Here's a game running alongside visualization tool. 

I'm walking around a room, and I'm looking at the raycasts and box casts our player 
movement is making 

I can look at it from the 3'^^ person view, I can only render the relevant data. 

I'm noticing something strange, there's a flurry of box casts sometimes. It might 
probably result in a framerate hitch. 

So I'm trying to reproduce this. 

I find a place where it's happening. 

And I'm flying around to see what causes it. 

So far, so good. By the way, I can play with that physics at any point, but it won't 
affect the game, so it's not useful in this case. 
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When That Is Not Enough 










Well, what if just looking at the problem does not present an obvious fix? 
What if we have to actually debug code? 













At least when you debug your own code, it's easy to see the parts. The components. 
What they are supposed to do. 

It's easy to check and see if they are actually doing it. 


36 









other People's Code 


Unfamiliar Code Looks Like This: 


Working in a team, it's different... It's hard to see the whole picture. It's hard to check 
if the parts are working properly. 
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other People's Bugs 


Bug in Unfamiliar Code Is Here: 



Even when I find a bug in foreign code, there are usually many ways to fix it and I'm 
never quite sure which one is the best. 

Or maybe I misunderstood the code and my fix will cause another bug. 
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Tools 




And the debugging tools we have are very basic: We have the Watch window, and 
printf. That's just inadequate. 





















Second Life of Testbed 



And now we have this wonderful application.. I wonder if we could use it when we step 
through the breakpoints just like we use it when we walk through the game. 

We need to leverage it! But how? 
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' pSimul*tionWorldTr*nsforms.l4 0x00000000579603*0 )po$=( 6904.11133, 6710.47217, 32563 

• ■ 101 po».( 690411133, 671047217, 325.638428) x.(0.772891462 

- - fnatnx3x4 t posx( 6904.11133, 6710.47217. 325.638428) xx(0.772891462 

• -m.flMatVal 0x00000000579603*0(0x00000000579603*010.772891462 0.3 

posx{-690L788S7. 671184180, 349J43170) xx<-0.02101&858 
pc»>< 689630273, 6697.04053. 322.380066) x*<0.372540712 
po*=( 6907.72754, -6725.45947, 321657361) x=(-0,086953304 
pot.( 690204639, 671183691. 364934204) x.J)0402162559 
po4x( 690062715, 671228955. 380.599396) xx(O.0578S26743 
p<».( 690003711, 6714 20801. 391.881500) x.(0,610547960, 
po»( 689298975, 671108350, 381J04016) xx(0.299724787. 
p<»x( 6899.02979, 6720.03516. 380205017) xx{0.S99S34O94. 
po«.< 6890.17285, 6721.56641, 397.602539) XM0.79821217L 
>] po5=( 6879.02051 671674902 393.418640) x=l0.1362S1017 

11 po«-( 688349463, 673069482. 392627655) x-(0.263435274, 

!| posx( 689262842 -672141406. 408810089) xx(-0.579993010 

- -1131 _ pot.< 6884 52002, 672323447, 396799957) x.(0.256165055. 


m.DabugNam* 
m pWorld 
m.pF«Mo<**» 
m riBMxdUndarRalax 
m_flStretchUnd«rft«<*x 


0x0000000002bdf700 |m Filt*f=(in GroupP8ifS=OxO0(»0C00O2 
0x00000000110384(9 (ni.nFI*9i.4009750271 m.nNodeCount 


Tt.nNodcCounl 

n_nP*xticl*Coonl 

n.pPartKies 


0x0000000011266300 po5x( 687629443, 6769.97656, 364 533 
0x0000000011266840 l.aiNDOOOO, l.aiNDOOOO, l.UNDOOOO 
0x0000000011266920 lalNDOOOO, l.aiNDOOOO, l.aiNDOOOO 


The usual debug experience consists of sifting through 
through lines of code, imagining our 3D world in our 


Watch windows, stepping 
mind's eye.. 



























Improved Debug Experience 


« m.pPa<ticl«.14 0>000aa00011266300 (pot>( &876J9443. 6769.976S6. 364. 

• • |0| pos-( 6876.29443. -6769.976S8 364.S33264) xs:(-0.231S917Sl 

• *111 p«.( 685443750, 669925000 336452576) x.(a23189175L 

• • |2| pos=( 6866.7251ft 6739.01172. 352239136) x=(023189175t 

> •131 poJ«( 6896.30566, 6721.00684, 412660950) «•( 0621897459 

> • |4| pos=( 6903.90430. 6718.92432.419238647) x=(-0.7950301i; 

• -151 pot.( 6892.56396 672199951. 408815186) x.( 0.576003432 

• * |6| po»={-6877.71094. 6774.56104. 366J53394) x=( 0231891751 

• • |7| pof( 6865.03955. 6733.55664. 350.073364) x.(02318917SL 

• • |8| po«.( 6915.52539, 6715.27734, 420.790619) x=( 0.745445311 

• - |9| po5.( 6862.48145, 6725.27881, 346.786865) x.(0231891751. 

- • |10| P0|.( 6925.21631, 6711.95996 417.005035) x-( 0.674079120 

> - 111) pos^l 6882.99365. 6791.65576 373.140533) x=( 0.231891751 

• *1121 pool 6887.22559. 6805.36816 378.716858) x>( 0.231891751 

> • |13| pos=(-6899.58836 -6845J72S6 394.600006) x=( 0231891751 

- pSitnol»tKinWofWTri«nformi.l4 0x00000000579603*0 |p«-( 690411133, 671047217, 325.63 

• |0| pos=(-6904.11133, -671047217, 325.638428) x=(0.77289146Z 

• - m»trcx3x4.t pof( 6904.11133. 671047217, 325.638428) x.(0.772891462 

• ' m flMatVal 0x00000000579603*0 10x00000000579603*0 I0.77289146Z 0.3 

"111 pot>( 6901.78857. 671184180, 349.343170) x>( 0021016858 

»|2| poix( 6896.30273. 6697.04053, 322J80066) xx(0l37254O7U 

" |3| pos«( 6907.72754, 672545947, 321857361) x«(-0 086953304 

‘ |4| pojx( 6902.04639, 671183691, 364.934204) xxt0.0402162S59 

»|5| pos.( 6900.82715. 671228955, 380.599396) xx(0 0578526743 

MSI pot>( 6900.03711. 6714.20801. 391881500) xx(ft610547960. 

> |7| pos=( 6892.98975. 671L08350. 381.304016) x=(0299724787, 

M8| pof( 6899.02979. 6720.03516 380205017) x«(0 599534094. 

|9| pos=(-6890.17285. 6721.56641, 397.602539) x=(0,79e212171. 

" 1101 p(»«( 6879.02051. 6716.74902. 393418640) x>Iftl36251017. 

• |11| poix( 6883.49463, 6730.69482. 392.627655) xx(0263435274, 

'1121 pot.( 6892.62842 672141406 408810089) x.( 0.579993010 

■|13|pos=( 6884.52002 -672383447, 396799957) xx;02561S5055. 


-m DabugNam* 

■MnjjWortd 

>Mn.pF*Mod*l 

- m.flBendUndetflctax 

» m.flStnxtchUndarRalax 
xm flOverPredict 
X m.nNodaCount 

- m nPartideCount 


Ox0000000002b<H700 (m.Fihar=|m.GroupPairsxOx0000000002 
0x00000000110384f9 (m.nFUgj.4009750271 m.nNodcCount 
0.000000000 
0.000000000 
0.000000000 
14 

0x0000000011266300 pmm( 6876.29443. 676997656. 364 533 
0x0000000011266840 l.#INOOOOO. LalNDOOOO, LtlNDOOOC 
0x0000000011266920 l.«NO0000, 14INDOOOO, I aiNCX)000 



Ideally, I want to see the world just like I did in the Physics Testbed, but I also want to see it 

in the Physics Testbed, but I 

I wialsi© i/veaMdQo^aiitoQhaiflgi^iQyiiJ^ wbtlttieoasnlistepritephiigh mytbraalipteimsii for me. 

That 

would be ideal for me. 

That would make debugging much more comfortable, enjoyable and 
productive. 





















Let me show you what I mean. 

As you can see, this is the place with too many box casts that I discovered in the 
previous demo. 

I place a breakpoint in the box trace function. Every time the it hits, I'm looking at the 
3D world. I'm seeing the very last box cast. 

This lets me step through bunch of casts, visualizing each of them in Physics 
Testbed. 

This whole thing happens inside of a frame. 

Please note that the game didn't send any data packets to Physics Testbed. It's 
casting boxes in a very tight loop. The testbed is effectively acting like a watch 
window. 
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Reading Memory Directly 



The method that lets us do it is ReadProcessMemory API. 

The Physics Testbed, acting in a way like a debugger, reads memory from the game 
without the game knowing it. I call this snooping. There is no way for the 
debugger to alter the game state, which is a nice benefit. 

It works when process is stopped in debugger 

There is no activity required from the game side besides a mutex and advertising 
where to start the search (the root data structure). Both are trivial to implement 
and are not intrusive. 
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ReadProcessMemory performance 



The biggest question when I thought of this idea was whether it's fast enough. 
So I wrote a small benchmark. 

It is slow to read every 4- or 8-byte field across the Process boundary 
Every call to ReadProcessMemory descends into kernel, performs a syscall — 
it costs you at least 1 us, limiting your bandwidth. 

But just copying 4 bytes in-process takes a thousandth of that time. 






















ReadProcessMemory performance 



I simply implemented software cache layer that reads at least 4Kb page every 
time you snoop a byte and caches that page 

You always know you can read the 4KiB page if you can read a single byte of 
it. There will be no memory protection faults. 

This cache helps a lot with reading C NULL-terminated strings, for which 
length is unknown until you read the whole string one byte at a time. 























Traversing Data Structure 



ReadProcessMemory 



So, we can snoop the bytes from the game physics, but how do we know what to do 
with them? 

In fact, walking the memory of the other process is very much like serialization. If you 
know your data structures, you can traverse them in another process using 
ReadProcessMemory just the same as you traverse them in the same process 
when you serialize them. 
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Snooper and Serializer 


C++ Classes 


Clang 


Snooper Code 



Serializer Code 


We can still use Clang to understand our data structures. 

We can use all our existing code as input and spit out the snooper routines. 




















Snooper: Code Generation 


C++ 


Clang 


C++ 


#inc1ude 


void CRnCapsuleshape : :Snoop( CRnSnooper*pln , 

class CRncapsu!eshape : 


const CRnCapsuleshape *pLocalCopy ) 


{ 

public CRnShape 

pub1 ic: 


CRnShape: iSnoopC pin, pLocalCopy ); 


m_flRadius = pLocalCopy->m_flRadi us ; 
for( int nElement = 0; 

private: 


nElement < 2; ++nElement ) 

vector m_vCenter[ 2 ]; 


{ 

float m_flRadius; 


: :Snoop( pin, 

auto_serialize; 


&pLocalcopy->m_vcenter[nElement] , 

}; 


m_vCenter[nElement] ); 

} ^ 


Just like with the serializer, we are transforming C++ code into additional C++ 
code for snooper 

Something is becoming clearer now. Generating code as a bunch of printf() 
statements would really be awkward. Because we now have to print out 
serialize, unserialize, snoop and possibly collect statistics routines. 
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Snooper: The Same Json 


C++ 


Clang 


json 


#i nc1ude "..." 

class CRnCapsu!eShape : 
public CRnShape 

pub1 ic: 

private: 

vector m_vcenter[ 2 ]; 
float m_flRadius; 
auto_serialize; 


"CRnCapsuleShape" : { 
"fields" : { 

"m_flRadius" : { 


"typeName" : 
} > 

"m_vCenter" : { 

"float", 

"c1assName" 

: "Vector", 

"arraySize" 
}, ^ 

"bases" : [ 
"CRnShape" 

: 2, 


So, we still generate the same json file as we did for serialization. 



























Snooper: Different StringTemplate 


C++ 


Clang 


json 


StringTemplate 


C++ 


void <name>: : Snoop ( CRnSnooper*pln , 
const <name> *pLoca1Copy ) 

<class.bases : {b | 

<b>::Snoop( pin, pLocalCopy );}> 

<class.f 1 elds.keys, cl ass.fields.values: 
{k,v I 

<snoop_field(name=k,props=v)> 

> 

<if(cl ass.postinitMethod)> 

AfterRestoreC pin ); 

<endif> 


The StringTemplate for the snooper looks like this. It's a different string template, but it's 
small. All my snoop code is generated from a 200-line template. These are a few of those 
lines. They auto-generate the code to snoop most classes in our physics engine. 

You feed it the same json file as for the serializer. It spits out snooper C-i-i- file. It's that simple. 
































Snooper: Generated Code 


C++ 


Clang 


json 


StringTemplate 


C++ 



void <name>: :Snoop( CRnSnooper*pln, 
const <name> *pLocalCopy ) 

<class.bases : {b | 

<b>::Snoop( pin, pLocalCopy );}> 

<class.fields.keys, cl ass.fields.values: 
{k,v I 

<snoop_field Cname=k,props=v)> 

> 

<if(cl ass.postinitMethod)> 

AfterRestoreC pin ); 

<endif> 



void CRnCapsuleShape: iSnoopC CRnSnooper*pln , 
const CRnCapsuleshape *pLocalCopy ) 

CRnShape: iSnoopC pin, pLocalCopy ); 

m_flRadius = pLocalCopy->m_flRadi us ; 
for( int nElement = 0; 

nElement < 2; ++nElement ) 

{ 

: :Snoop( pin, 

&pLocalCopy->m_vcente r[nElement] , 
m_vcenter[nElement] ); 

AfterRestoreC pin ); 


This here is the template and the resulting generated C++ code for snooper. 

I'm snooping the whole struct into a local copy. Then I read out all the plain data 
elements. 

All the non-plain data elements are recursively snooped by other overloaded Snoop 
functions. 

The template is smart enough to know where I have arrays, templates, pointers to 
follow and so on. 
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Full Pipeline 


C++ 


Clang 


json 


StringTemplate 


C++ 


#i nc1ude 


ASTConsumer 


{ 


void <name>:: 


void Shape: : 

"physics.h" 




"fields": ... 

} 

— 

SnoopO ... 

— 

SnoopO ... 


Just to bring the point home^ here's how the full code parsing and 
generation pipeline looks in our case 

Parsing C++ is probably the most intimidating thing in this case. But after you spend a day or 
two learning Clang's API, you might find it rather intuitive. 
















































Example: Snooping Joint Stack 


• 1 u j- 

PnKcu; ! (S7ao| phyiKs.t«tt>ed.<n 


► Continue • Aoto -! O * 

4 - Threed [^IMwThnl 





CRnWand Proj^temt^ 

' tl** bool Cltnhl<nd::Prc)KtJ«ntt< nJoiMCoum) 


U -» CRnbUnd 

*1® Pn>fectk»wit<inC rUointCounI) 



} 

//-■ 


348 -bool CRnisland::ProjectJoints( int nJointCount ) 

_ 349 { 

350 [I bool bAllJointsOK = true; 

351 for ( int nJoint = 0; nJoint < nJointCount; ++nJoint ) 

352 { 

353 CRnJoint* pJoint = m_Joints[ nJoint ]; 

354 

355 pJoint->Project(); 


return bAllJointsOK; 




Let's see a little demo of what we have as a result. 

I configured a projection-type solver of a chain of hinge joints. As you can see, small 
chains solve fine, but larger chains start to break constraints. 

I wanted to see how the solver behaves inside the iteration loop. 

So I put a breakpoint where each joint is solved. 

After each breakpoint hits, I switch to the Testbed and snoop. I have two testbeds 
here, one debugging the other. Notice that one is unresponsive - that's the one 
that I'm debugging. And another one acts like a watch window. 

So what we have here effectively a visualization of the internal loop of a projection 
constraint solver. 
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Fixing stuff, whether it's yours or not, is unavoidable. At least if you make anything 
complex. 

The faster you can drill down to details, the more efficient you are while debugging 
problems, the more time you spend developing and not fixing stuff. Once you 
make debugging 10x faster and more convenient, there's a quantum leap : you 
can develop features with complexity that would otherwise be infeasible to 
develop. 

That's why it's important to make the process as seamless and convenient as 
possible. It pays with higher productivity in the end. 

There are a few more technical details, though 







Divide and Conquer 



Our Testbed is a tool for divide-and-conquer debugging. It makes it quick and 
painless. 

For example, we have a collision filtering system, and time to time something is not 
colliding right in the game. We just fly around, click on objects that don't collide 
and drill down to their flags. In most cases we see exactly what's happening 
within seconds, and you don't have to be a programmer to do that. 

Another example: we had an incredibly slow physics. It worked fine, but was very 
slow. All contacts looked correctly. So we snooped and clicked the objects that 
were awake. Their contacts looked fine. It took us some seconds to realize there 
were too many contacts on them. 
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Mutex in game loop 



Snoop 


t_I 

Mutex 


One consideration here is thread safety 

While you cannot use the Watch window in VS IDE while the game process is 
running, you can snoop. 

Use Mutex to guard the physics step routine 








































SEH 




try { . . . } except (){} 

1 Snoop 1 

I 1 

1 Mutex 1 






Snooping a running process is tricky: data structures are in flux while you are 
reading them. 

It's impossible to crash the game by reading data from it. But the snooper 
might crash if you snoop an inconsistent memory. I'm just using 

(_^try..._except) to catch that. Strictly speaking it's not a safe solution, but it 

makes the experience smoother. 


















SEH != try...catch 


_try { 


try { 

} except(){ 

1 = 

} catch(...){ 


■ 


} 


} 


*** Also, neither is necessary 


Just to be clear 

Structured exception handling is not the same as C++ exception handling. 

This goes outside the topic of this talk. 

In any case, you don't really have to use either of these. 

I just find it nice to wrap my sloppy debugging tool code in case I happen to snoop 
bad data structure. 

If I released it as a commercial product, I'd expend the extra effort to detect that 
memory is unreadable and not crash. 













When the game is stopped on a breakpoint, just ignore the Mutex. Or time it 
out. 

Data structures cannot change when the game is frozen, so there's a high 
chance snoop will succeed if they are consistent. It depends on where you put 
the breakpoint. 
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Clang Annotations 


# define CLANG_ATTR(ATTR) \ 

_attribute_( ( annotateC attr ) ) ) 

#define SERIALIZE_ARRAY_SIZE( SIZE ) \ 

CLANG_ATTR( "array_size:" #SIZE ) 


class CSomedass 

{ 

uintlG *m_pNodeToCtr1 

SERIALIZE_ARRAY_SIZE( m_nNodeCount ); 
uintlG m_nNodeCount; 


bool HasAnnotatedAttrC 

const clang::Decl *pDecl , 
const char * pSubstr ) 

if( clang:lAnnotateAttr *pAnnotate = 
pDecl ->getAttr<clang: : AnnotateAttr>() 


ShortStringRefVector attrs; 
pAnnotate-> 

getAnnotationO.splitC attrs, " " ); 
return std::find( attrs . beginQ , 

attrs. end(), pSubstr ) != 
attrs. end() ; 

return false; 


Sometimes we need some annotations to our code. E.g. for each pointer you 
serializer will need to know how many objects it's pointing to. It's easy to put 
this in (see the left slide). 

And ot's easy to add and read those annotations in the parser ( see the right 
slide) 
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Complex Case: Polymorphism 


Struct Vtab1eRecord_t 
{ 

template <typename t> 

void lnit( CUtlStringToken name ) 

T* pobiect = new t; 

m_nVTaDle = *( ( uintp* )pObject ); 

delete pObject; 

m_nClassName = name.m_nHashCode; 

} 

uint64 m_nVTab1e; 
uint64 m_nClassName; 

}; 


I'd like to talk about a couple of less trivial cases, like classes with Vtables. 

I auto-generate a function that creates an instance of each class with Vtable, 
and copies its Vtable point into a known place (an array of uintptr_t). 

The snooper can then read the array and use it to recognize class type by its 
vtable pointer. It's like RTTI that works across process boundaries. 








Complex Case: Polymorphism 


void initvtableRecordC CUtlVector<vtab1eRecord_t> *pTable ) 

<data.cl asses.keys, data.cl asses.values: {k,v | <register_class(name=k,class=v)> }> 


register_class(name,class) ::= « 

<if(cl ass.isLeaf)> 

( *pTable )[ pTable->AddToTail () ]. ini t\«name>\> ("<name>" ); 

<else> 

<if(!cl ass.isBase)> 

// <name> is neither leaf nor base, no need to register it for auto-recognition 
<endif> 


This is how I generate that code with StringTemplates 








Complex Case: Polymorphism 


void InitvtableRecordC CUtlvector<VtableRecord_t> *pTable ) 

( *pTable )[ pTabl e->AddToTail () ].lnit<CRnweldDoint>( "CRnweldloint" ); 

// CRnshadowcontroller is neither leaf nor base, no need to register it for auto-recognition 
( *pTab1e )[ pTabI e->AddToTail () ].lnit<CRnSpringJoint>( "CRnSpringJoint" ); 

} 


And this is what the generated code looks like. One line is generated for every 
class with vtable. It’s done automatically, so you don’t have to remember to 
add a class to the list once you add a virtual. 

You don’t have to remove a class when you remove the last virtual from it 
either. 
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More Complex Cases 


Compound data structures 


Automated CodeGen 

. World Root 


These data structures change frequently 

• Rigid Bodies 


Easy to generate code for 

• Joints 


Easy to mark up with clang annotations 

. Contacts 



. etc. 




Primitive data types and special cases 


Write code by hand 

. float, int and other PODs 


Change infrequently 

. Simple types without pointers 


Sometimes hard to generate code for 

. std::vector<> and external containers 


Building blocks for compound data 

. Special containers 




For some data structures, it's just easier to write serialization code by hand. The 
code generator should know about leafy data types like int and float, and 
generate serialization code automatically. There's no problem with structures only 
consisting of those data types, those can be simply copied directly from another 
process address space. 

It's a bit more involved with pointers. In general, C++ does not provide enough 
semantics for you to know if a pointer points to one object, or a vector of objects. 
I'm using clang annotation to add that semantics. So, the code generator looks at 
that annotation and computes the size of the array and serializes accordingly. 

We don't use multiple heaps for our physics engine, but if you do, you'll have to 
provide semantics about how to allocate the snooped or unserialized data 
structures. 

In strange cases, like an int that is a pointer, but the lower 2 bits have a special 
meaning, you probably want to write that routine by hand. Also when you have a 
union and have to run code to decide what it means. 

You have a choice whether to make your code auto-generator smarter (and harder 
to maintain) or snoop a specific class manually. It's a judgement call. 
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Statistics 


C++ 


Clang 


json 


StringTemplate 


C++ 


Non-trivial data: 

46 classes 
320 fields 


***AII generated code is typo-free 


We currently have 46 non-trivial classes with 320 fields for which serializer and 
snooper code is generated. 
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Statistics 


C++ 


Clang 


json 


StringTemplate 


C++ 




Parser: 









1000 lines 








***AII generated code is typo-free 


The parser that uses clang is about 1000 lines long. I probably spent a couple 
of weeks part time writing it, and I've been using it for a year. 
















































Statistics 


C++ 


Clang 


json 


StringTemplate 


C++ 


ion-trivial data: 




Generated data: 


Sr tc:. lul^iie: 


inoop con*-: 

500 ‘inr- 

classc 

320 fie- i- 


lOOC lines 


2800 lines 


- ■ temp!^^t« 

^' 0 lines 


n. :i-n i^e code: 
^:.00 i..*,- 


***AII generated code is typo-free 


The generated json is 2800 lines long 
















































Statistics 


C++ 


Clang 


json 


StringTemplate 


C++ 






Snoop template: 

200 lines 







Serialize template: 

400 lines 




***AII generated code is typo-free 


Snooper template is 200 lineslong . 
Serializer template is 400 lines. 
















































Statistics 


C++ 


Clang 


json 


StringTemplate 


C++ 










Snoop code: 

2500 lines 









Serialize code: 
5600 lines 


***AII generated code is typo-free 


Auto-generated snooper is 2500 lines long. 

Serializer is 5600 lines long, because it includes 3 directions: serialize to 
stream, unserialize from stream, count number of bytes used (very similar to 
Serialize, but actually counts allocated memory). 

It takes 2-8ms to snoop a typical game frame, which is normally maybe 10 mb 
large 

















































Every game physics engine eventually gets integrated into a game. It makes 

change-compile-test iteration much longer. Especially when artists start building a 
huge map, and new exciting bugs show up that only happen in this huge map 
once in a blue moon. 

It makes bug hunting mind-numbingly slow. Because the game code is much, much 
larger than the physics code we know so well. 
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Physics in Games: Perspective 



Game code is written by many people over many years. Physics engine is self-contained 
and neat, almost miniature in comparison. 

At valve, the sheer amount of logic in the old game code with history is very large. 
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Physics in Games: Perspective 



It's really easy to lose perspective when you work on physics for a long time. 

The ultimate goal of physics in a game is to make the game better. It's not to make the best 
or fastest or true to life physics simulation. That is why in-game debugging and 
optimization is very important. 

If the game wants to cast a lot of rays, we need to optimize that first. Games don't generally 
have stacks of cubes everywhere, so that's not the best benchmark for a game physics 
engine. It's best to put it in the game and debug and profile it in the game, if you want it 
to perform well in the game. 
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Game Data Visualization 



Run Generating_Nav 

Here's another example of an unexpected benefit of visualizing everything in physics engine. 
Testbed records and displays all traces. And Ieft4dead navigation system does a lot of 
traces. So, when connected to the game, Testbed visualizes an Al algorithm. 
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I suggest we can all spend a couple of weeks up front to make our life more 
comfortable and deliver a better product. Sometimes you can't do that, but 
generally if your time horizon is years, you can afford a couple of weeks. 

What I'm describing is pretty general technique. Game physics is just an example, 
but it's usable in any complex software. 
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Debugging Visually 


q Ts p _ O Gi- 

O • C H J* ► £«»«. . rittP, ■ll'I, 



[:,. , ' -imjiIO .nt 
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Eventually, I want my Visual Studio to look like this when I'm debugging. 

You can make a debugger extension with VS Extensibility, and I might make one 
some day. But when I experimented with it, it was very inconvenient to write a VS 
extension of such complexity. 
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AutoExp.dat, *.natvis 


[AutoExpand] 

Vector =x=<x,g> y=<y/g> z=<z,g> 

Quaternion= x=<x,g> y=<y,g> z=<z,g> w=<w,g> 

[Visualizer] 

CStrongHandle<*> { 

preview ( $c.m_pBinding->m_Name->m_ResourceNameSymbol.u.m_pAsString ) 

children ( 

#{ 

Data: ($T1 ^)($c.m_pBinding->m_pData), 

[raw members]: [$c,!] 

) 

) 

} 


I hope everyone knows about autoexp.dat. It lets you visualize in text, so to say. 

I strongly suggest you just put all your data structures in there. Try to make them 
more readable - in your Watch window. It helps a lot. 
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Organizing Data Structures 



A word about organizing your data strucutres. 

It's most convenient to have a root object from which you can crawl all your data. 

I just gather all the globals in one object, and only publish a pointer to that one object 
to the snooper. 

Physics will have some data structures. You'll parse and process them. 

You'll probably have some external pointers, like pointers to the vertex and index 
buffers for debug drawing. 

You can write routines for snooping and serialization of those things manually 
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Out-of-Game Visualization 



Another thought. Visualizing in an external app doesn't have to use 
ReadProcessMemory API. 





























Using TCP/IP 




All the leading physics engine have some sort of over-TCP/IP visualization. 

Unfortunately it doesn't work when the game is stopped in debugger, frozen or 
crashed. Unless you stream your game state all the time, and that spends CPU. If 
you don't know where the problem is, and stream a lot of detailed data, it really 
takes a lot of CPU. And you have to remember to turn it on. 

But it works on remote machine. So it's a viable option. 
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Using Shared Memory 


Game 



Shared Memory 
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An alternative to TCP/IP that works when the game is stopped in debugger ( or 
frozen or crashed ) would be to use shared memory. 

Win32 API allows processes to share memory directly. One way is to use a memory- 
mapped file, but you need to dedicate a region of memory for sharing, and some 
of the memory would be passed down from the game (like pointer to collision 
resources), and it would be very inconvenient to make sure all of that is allocated 
in the shared memory region 
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Hopefully we'll eventually be able to save/load/rewind the whole game with this 
method. Our game code base is much bigger than physics engine though, so 
that's a lot of work. 
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Serialization 



class Contact_SerializGd 
{ 

int m_nshapelndex[ 2 ]; 
int m_nNextlndex[ 2 ]; 

}; 


10010110010110 ... 


We are using the same framework to generate serialize/unserialize code. 
Every time we change our data structures, we just re-run the parser/generator, 
and it's all updated 

We serialize/unserialize a byte stream. But another variant would be e.g. to 
generate a reflected version of all your data, with indices instead of pointers. If 
you have reflection API, it will let you version the serialized data 
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Techniques: The Summary 


Technology 


Technique 

• Clang 


. Detailed Statistics 

. String Template 


. Serialization 

• Read ProcesMemory API 


• Streaming 



• Snooping 



• etc. 


To summarize, we were talking about a few uses of two technologies: Clang 
and String Template, and one simple Windows API: ReadProcessMemory. 
Together, these technologies can enable very powerful and complete 
serialization, an interesting debugging technique I call snooping, gathering 
detailed statistics without tedious coding and without that code going stale the 
next time someone adds a new array somewhere. 

The same serialization template can also be used to send data over TCP/IP, 
and/or serialize into an alternative data structure with reflection API (e.g. into a 
series of dictionaries) that can be saved and loaded into newer versions of the 
engine, and many more uses. 

It is possible to use snooping to read and display every frame in your game 
You will probably skip some frames unless you take care to synchronize 
You will probably waste time at the mutex, for the game has to wait for the 
snooper to finish, and it's less efficient to snoop than to pack local game data 
and send it through a pipe or socket. 

It is much faster to let the game actively serialize the world delta every frame 
and send it over the wire. It will also guarantee no skipped frames. 

It is easier to just reuse existing serialization and send every frame over the 
wire 

For a quick fix, just see the biggest pieces of data (probably mesh 
descriptions) and cut them out of the stream if you already sent them 
When visualizing streamed physics, I tied the Vbs/IBs directly into the world 
data structures. 

This automatically recreated Vbs/IBs on every streamed frame 

To fix that, I had to cache those debug objects off and have special callbacks 

reuse them. I did it with clang's annotations. 
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Pros & Cons 


Pros 


Con 

• Reliable Process 


• Implementation 

• Repeatable Debugging 

• Human Error Excluded 
. Neither Tedious nor 


Complexity 

Outdated Code 




Full-world serialization of this type includes everything, and has very high 
reliability. You don't have to worry about missing a thing or two: the 
parser/generator is unlikely to make a human error. 

This is usable to dump suspicious game frames for later examination, or 
streaming them. When debugging later, it's very much like debugging in-game, 
but it's repeatable and much more convenient. 

I don't like manual non creative work, and serialization code is always very low 
in creativity. Getting rid of it makes iteration faster and doesn't distract me from 
the creative process. 

Among the drawbacks, this method is more complex to start using it. You can 
still use the old methods in parallel with this method. 
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Make work fun! It pays to do that. 
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